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Creating Effective Surveys 


Best Practices in Survey Design 


Purpose 


When education leaders utilize survey results to inform their decisions, they are often left wondering if 
the “right” survey questions were asked. To help state and local leaders improve existing instruments 
or create better instruments for data collection, this quick reference guide highlights best practices in 
designing effective surveys. Administering better surveys leads to more specific and accurate data, 
and better data leads to better evidence with which to inform decisions. 


This guide focuses on four areas of survey development: 


«? general best practices in survey design, 
«P common problematic survey questions (and how to avoid them), 
« Likert scales and rating scales, and 


«> creating more inclusive surveys. 


Intended Use 


This quick reference guide is designed to help state and local education leaders and staff who are 
asked to develop or modify surveys but may not have a background in survey design. This guide 
will help you gain a more critical lens when contributing to survey design and development or when 
reviewing survey results. If you'd like more details on survey design and other survey-related topics, 
please refer to the list of resources at the end of this guide. 


https://ies.ed.gov/ncee/edlabs/regions/west/ 
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General Best Practices in Survey Design 


1. Define a clear, attainable goal for your survey 


What are you trying to learn about from your survey? Start by crafting a goal before writing any questions. 


Unclear, unattainable: “This survey will help us learn about everything parents are thinking about their child's 
math homework assignments.” 


Clear, attainable: “This survey is designed to understand parents’ perceptions about the difficulty, frequency, and 
variety of their child’s math homework last year. Responses will be used to shape the school’s guidance to math 
teachers in the coming year.” 

2. Don’t let your survey get too long 

Long surveys lead to fatigued or annoyed respondents so they become less likely to provide truthful, detailed 
answers or, more importantly, finish the whole survey. 

3. Keep it brief, simple, and specific 

Keep your survey brief and focused specifically on the exact data you wish to analyze. Ask only the questions 
needed to achieve your survey goal, and ask them as clearly and simply as possible. 

4. Save open-ended, challenging, and more personal questions for the end 

Allow respondents to get comfortable with the survey by asking easier, more general, and less personal 
questions upfront. Asking open-ended, challenging, and personal questions right away might feel jarring 

or intrusive to a respondent. 

5. Allow respondents to respond “not applicable (N/A)” 

Whenever possible, include the option to indicate that a question does not apply to a respondent's experience. 
Otherwise, respondents might skip such questions, leaving you to wonder why. 

6. Include a short introduction and a time estimate 

Before asking respondents to answer questions, help them feel comfortable and able to accurately estimate their 
commitment by welcoming them to your survey and indicating a realistic amount of time in which to complete it. 
7. Test or pilot your survey and survey platform beforehand 


Ask colleagues to test your survey first and give you feedback on clarity, ease of responding, and so on. 
This applies to both online and paper-and-pencil surveys. 
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Common Types of Survey Questions 


Closed-ended questions include multiple-choice, rating-scale (e.g., rate from 1-5), and checkbox 
(e.g., true/false) questions. These types of questions provide quantitative data and, generally, 

are easier for respondents to answer. Including more closed-ended questions than open-ended 
questions in your survey will reduce fatigue for respondents and increase the likelihood that they 
will complete it. 


Open-ended questions ask respondents for feedback in their own words (i.e., free response) 
and can provide rich, qualitative data. Because open-ended questions can take much longer to 
answer and analyze, include fewer of them and place them at the end of the survey. 


Common Problematic Survey Questions (and how to avoid them) 


Poorly written survey questions will reduce the quality and quantity of responses. Here are some common types of 


problematic survey questions to avoid as you craft your data collection instrument: 


Leading question A question that signals, Were your friendly library | Ona scale from 1 
prompts, or encourages staff members helpful as (completely unhelpful) 
a certain answer you engaged in summer- to 5 (very helpful), rate 
school teaching? the helpfulness of library 


staff members as you 
engaged in summer- 
school teaching. 


Loaded question A question that includes How much will test On a scale from 1 
an unjustified assumption scores improve because (decrease significantly) to 
or forces respondents to of your school’s new 5 (increase significantly), 
agree with an assumption reading program? how do you expect 


test scores to change 
because of the school’s 
new reading program? 


https://ies.ed.gov/ncee/edlabs/regions/west/ 


Double-barreled question A question that asks for 
an opinion about two 
different items but allows 


for only one response 


How do you think 
students’ test scores and 
attendance will change 
because of the new after- 
school program? 
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Separate into two 
questions: 


1. How do you expect 
test scores to change 
because of your 
school’s new after- 
school program? 


2. How do you expect 
attendance to change 
because of your 
school’s new after- 
school program? 


Double-barreled answer An answer option that 
presents two possibly 
different opinions as a 


response to one question 


What was your 
personal experience 
with mathematics 

in high school? 


Rate from 1 (did not 
like/did not succeed) 
to 5 (passionate about/ 
excelled at). 


What was your 
personal experience 
with mathematics 

in high school? 


Rate from 1 (negative) to 
5 (positive). 


Double-negative question A question that contains 
two negative elements 
intended to create a 
positive element, which 


can confuse respondents 


Is it not uncommon for 
teachers to coach a sport 
after school? 


How common is it for 
teachers to coach a sport 
after school? 


https://ies.ed.gov/ncee/edlabs/regions/west/ 
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Likert Scales and Rating Scales 

Closed-ended questions that incorporate a Likert scale or a rating scale are used frequently in surveys. When 
written well, such questions are generally less exhaustive for respondents to answer and are an excellent source of 
rich, quantitative data. Keep the following tips in mind as you craft your scale questions. 

1. Generally, five rating choices are enough 

Craft scales that yield only five possible answers (e.g., rate from 1 to 5, choose from five options). Provide more 
rating choices only if absolutely necessary to gather more detail. 

2. Maintain balance and objectivity 

Make sure that each scale provides ratings that are distributed to provide an equal number of choices with a 
positive or negative connotation. 

3. Offer a neutral option 


Include a neutral option to identify whether respondents feel neither positive nor negative about a topic. Allowing 
them to express indifference, rather than skipping the question or answering “not applicable,” will give you a more 
accurate representation of their perception. The neutral option should be in the middle of the scale. 


4. Don't forget to provide a “not applicable (N/A)” option 


Enable respondents to opt out of responding by indicating that a question does not apply to their experience. 


In the following example, the rating scale on the left is improved by changing the options to those on the right in 
order to ensure balance and objectivity and to provide a “not applicable” option. 


Needs Improvement Improved 

1. Very helpful 1. Very helpful 

2. Helpful 2. Helpful 

3. Somewhat helpful 3. Neither helpful nor unhelpful 
4. Neither helpful nor unhelpful 4. Unhelpful 

5. Not helpful 5. Very unhelpful 


6. Not applicable 
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Using Scale Questions to Understand Change 


Scale questions lend themselves well to understanding changes in opinions, perceptions, or 
knowledge when asked, as identically as possible, both before and after an event. A pre- and 
post-event survey can be completed either (1) in two phases, before and after an event, or (2) one 
time after the event. Individual or aggregated responses can then be compared to understand 
whether changes in opinions, perceptions, or knowledge have occurred. 


Pre-event question: Please select the option that best describes your level of 
understanding of the topic(s) before you attended the webinar. 


Post-event question: Please select the option that best describes your level of 
understanding of the topic(s) now that you have participated in the webinar. 


Options: (a) none at all, (b) a little, (c) a moderate amount, (d) a lot, (e) a great deal 


CHANGE IN UNDERSTANDING - HYPOTHETICAL EXAMPLE 


Level of Understanding Before and After Attending Webinar 


ee 83% 


20% 30% 40% 50% 60% 70% 80% 90% 


Note: Percentages indicate the proportion of 
participants who reported they knew “a lot” or 
“a great deal” about the topics presented in 
the webinar before and after attending. 


Before After 
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Creating More Inclusive Surveys 


It is essential to consider the various identities and lived experiences of respondents. Ensure that all survey language 
is inclusive for all respondents. Otherwise, they might be less likely to provide honest answers or to complete your 
survey altogether. Consider the following suggestions as you design your survey questions. 


1. Be thoughtful about demographic questions 


Collect only the demographic information you need. When doing so, offer your respondents flexible and 
robust options for identifying themselves. For example, demographic questions on gender should include a 
spectrum of potential gender identities such as male, female, transgender, non-binary/non-conforming, and 
prefer not to respond. 


2. Make survey questions mandatory only if a response is necessary 


Unless a response is imperative to utilizing the data, do not make questions mandatory. Requiring an answer to every 
question can make respondents feel uncomfortable or unwilling to complete the entire survey if they would rather not 
respond to one or more questions. 


3. Be mindful of language used in your survey 


Consider the inclusivity of your language and how it might be perceived by all respondents. Think about relevant 
context, history, and the implications of how statements are phrased. Don’t include language that ascribes or 


[! 


assumes a certain perception or viewpoint as something that should be “standard” or “normal” to your respondents, 


as it is possible that a different viewpoint might be the common understanding in their lived experience. 


4.Consult resources on inclusivity and bias-free language 


If you are unsure, the resources included at the end of this guide can help you check that the language of your 
survey is inclusive and bias-free. 


https://ies.ed.gov/ncee/edlabs/regions/west/ 
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